Extended Performance Graphs for Cluster Retrieval

Performance evaluations in Probabilistic Information Retrieval are often presented as Precision-Recall or PrecisionScope graphs avoiding the otherwise dominating effect of the embedding irrelevant fraction. However, precision and recall values as such offer an incomplete overview of the information retrieval system under study: information about system parameters like generality (the embedding of the relevant fraction), random performance and the effect of varying the scope is badly missed. In this paper three cluster performance graphs are presented. In those cases where complete ground truth is available (both cluster size and database size) the Cluster Precision-Recall (Cluster PR) graph and the GeneralityPrecision=Recall graph are proposed. In those cases where cluster sizes are unknown (and so recall) the double logarithmic Cluster Precision Window graph is proposed. 1 Shortcomings of presently used retrieval performance measures Performance characterization of content-based image and audio retrieval often borrows from performance figures developed over the past 30 years for probabilistic text retrieval. Landmarks in the text retrieval field are the books [12] and [11] as well as the proceedings of the annual ACM SIGIR [7] and NIST TREC [14] conferences. In the area of probabilistic retrieval the results of performance measurements are often presented in the form of Precision-Recall (or Recall-Precision) graphs and PrecisionScope graphs. Each of these standard performance graphs provides the user with incomplete information about how the IR System will perform for various cluster sizes and various embedding sizes. Generality (influence of the relevant fraction) as a system parameter hardly seems to play a role in performance analysis. Although generality may be left out as a performance indicator when competing methods are tested under constant generality conditions, it appears to be neglected even in cases where generality is widely varying (a wide range of cluster sizes in one specific database is the most frequently encountered example). That generality for a cluster of relevant items in a large embedding database is often ≈ 0.0 does not mean that its exact low level no longer matters. A continually growing embedding around a constant size cluster of relevant items will eventually lower the overall precision-result curve (for the user) to unacceptable low levels as is shown in Figure 1.